The Decline and Fall of Type II Error Rates

Introductory statistics students learn that hypothesis tests in­ volve two types of error. If we reject a true null hypothesis, a Type I error occurs. If we fail to reject a false null hypothesis, a Type II error occurs. Students become adept at using tables to find critical values that control the probability of a Type I error. However, the calculations needed to control the probabil­ ity of a Type II error can be complex and typically receive less emphasis in introductory courses. This is unfortunate as the sci­ entists and engineers who take these courses and later design ex­ periments sometimes fail to perform proper power calculations (power = 1 Prob(Type II error)). Instead their sample sizes are sometimes based on past practice or on available resources. As a result theirexperiments can be underdesigned (sample sizes are too small) or overdesigned (sample sizes are unneccessarily large). See Lenth (2001) for an overview of sample size issues. In one sense a researcher does not design an experiment to achieve a certain Type I error probability. Instead, at the anal­ ysis stage, the researcher just enters a critical value table via the targeted probability and the appropriate degrees of freedom. Given that the scientist’s model and distribution assumptions hold, the scientist is assured that the desired Type I error prob­ ability will result. Thus an analysis of Type I error is relatively easy to incorporate into an introductory statistics course or into a plan of experimentation. On the other hand, achieving a desired level of Type II error requires much more forethought. Prior to performing an experiment, a researcher must specify: